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ABSTRACT 

Although studies have 

statistically sophisticated prediction schemes for use in screening 
or placing clients in rehabilitation programs, they typically lack 
comparison with simpler methods, e. g. , the single best predictor 
method. In this study of 296 disabled clients referred to the St. 
Louis Jewish Employment Service for vocational evaluation, such a 
comparison was made. Variables included race, sex, age, education, 13 
W'echsler Adult Intelligence Scale subtest and total scores, and 10 
ratings of workshop performance. Employment status, assigned to the 
evaluated clients, was a further seven-categorized variable. Three 
prediction techniques were employed: (1) the multiple linear 

regression technique, (2) the multiple linear regression of factor 
scores, and (3) the single best predictor method., A cross-validation 
sample was utilized. The methodology was described and the results 
discussed. Two points were demonstrated: (1) the most useful 

prediction model may be the least statistically sophisticated model; 
and (2) shrinkage in predictive power upon cross-validation may be 
considerable, especially when regression models are used. Possible 
of prediction schemes were considered. (TL) 
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Many attempts have been made to devise prediction schemes to be 
used in screening or placement of clients in various rehabilitation pro- 
grams (Ayer, Thoreson, Butler, 1966; DeMann, 1963; Drasgow and Dreher, „ 
1964; Norris, Marra, and Zadrozny, 1960; Perlman and Hylbert, 1969). 
Although these studies typically employ sophisticated statistical or 
configural techniques in developing prediction methods, they fail to 
compare the developed methods with simpler methods, e.g., using base 

rates, using the single best predictor, etc. In addition, some studies 

j 

fail to crossvalidate prediction equations, and thus fail to take into 



account the shrinkage in predictive accuracy that almost always occurs 

j 

upon crossvalidation. The purpose of this paper is to compare two sta- 

I 

tistically sophisticated prediction models with a simple model, to demon- 
strate the effects of the shrinkage phenomenon upon crossvalidation of 
each model employed, and (finally to discuss the possible uses of such 



prediction schemes. 
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The subjects for this study were 296 disabled clients of the St. 
Louis Jewish Employment Vocational Service (JEVS) who had been referred 
for vocational evaluation and who had complete data available on the 
variables of concern. The variables of concern included 27 predictor 
Variables: race, sex, age, education, 13 WAIS subtest and total scores, 

and ten ratings of workshop performance. The workshop ratings were ob- 
tained from workshop counselors and supervisors after the client uad 
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completed three weeks of evaluation. Ratings were made on graphic rating 

i 

£ 

scales in the following ten areas: productivity, ability to get alonjjj 

with others, motivation, ability to follow instructions, punctuality,; 

| 

reliability, judgement, social competence and communication skills f ' 
cooperativeness, and dress and appearance. The. criterion, employment 
status, was a seven-categoried variable which was determined by the' 

1 

counselors after the evaluation program. ’ Category one was assigned to 
clients who became competitively employed; category two, to clients whom 
the counselors judged were employable; category three, to those who were 
judged to need educational training; category four, to clients who were 

I 

judged to need further work adjustment training; category five to those 
who were judged as employable only in sheltered settings; category six, 
to those who required further medical rehabilitation (e.g., psychotherapy); 

r 

and category seven was assigned to those who were judged to be unemployable 
regardless of additional service. 

First the 296 clients were randomly assigned to sample A (n = 151) 
or sample B (n = 145) . Sample A was the sample used to develop the dif- 
ferent prediction methods, and sample B was used as a ctossvalidation 
sample. A stepwise multiple linear regression analysis of sample A (Veldman, 
1967) was computed employing the 27 aforementioned variables as predictors, 
and the employability status variable as the criterion. The nine best 
predictors, i.e., those that contributed most of the variance to the 
predictor-criterion relationship, were selected. The selected predic- 
tors in the order of their relative contribution in predicting the cri- 




terion were: 



1. The rating of Motivation 

2. The rating of Productivity 

3. The rating of Cooperativeness 

4. The level of Education 

5. The WAIS Picture Arrangement sub test 

6. The WAIS Picture Completion sub test 

7. The WAIS Comprehension subtest 

8. The rating of Ability to Follow Instructions 

9. The rating of Dress and Appearance 

Three prediction techniques were used. The first method employed 
was the multiple linear regression technique. The nine selected pre- 
dictors were used to predict the criterion for sample A. Regression 
weights and the cutoff score minimizing the number of errors for the 
predicted scores were determined (Helmstadter , 1964). Predicted 
scores for sample B were computed using the regression weights from 
sample A. By using the cutoff: score from sample A and by comparing 

I 

the predicted score with the criterion, the number of hits and misses 
in prediction were determined for both samples. 

The second approach, multiple linear regression of factor scores, 
first necessitated a factor analysis of the nine predictors for sample 
A. Three factors were extracted and factor scores were computed for 
each of the factors (Veldman, 1967). The three factor scores then 
became the predictor variables in a multiple linear regression analy- 
sis of sample A. The remaining procedure involved determining the 
regression weights and cutoff score for sample A, applying these values 
to sample B, and counting the number of hits and misses in prediction 
for both samples. * 
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Finally, the single best predictor, counselors’ ratings of client 
motivation, was used in predicting the employment status criterion. A 
cutting score which minimized errors in prediction was computed on 
sample A and used on sample B. The number of hits and misses was also 

determined using this prediction technique. 

/ 

/ 

RESULTS 

The results comparing the three prediction models are presented 
in Tables 1 and 2. Table 1 shows that the technique using the single 
best predictor had the fewest number of total hits or correct predic- 
tions in the initial sample (sample A). However, on the crossvalidation 
sample (sample B) this technique had the greatest number of total correct 

predictions. 

Table 2 shows the validity rate, base rate, improvement over the 
base rate, and shrinkage upon crossvalidation for each of the three 
prediction models. In terms of both improvement in prediction over the 
base rate and shrinkage in predictive power upon crossvalidation, the 
single best predictor model, was superior to the regression of factor 
scores model which, in turn, was superior to the regression of raw 
scores model. 



DISCUSSION 

The results demonstrate two points: 

1. The most useful prediction model may be the least statistically 
sophisticated model. 

2. Shrinkage in predictive power upon crossvalidation may be con- 
siderable, particularly when regression models are used. 
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TABLE 1 

Comparison of Prediction Models 
on the Number of Hits and Misses 
for Successful and Unsuccessful Clients* 



SAMPLE 

Initial Crossvalidation 



MODEL 




Success 


Fail 


Total 


Success 


Fail 


Total 


Multiple Linear 


Hits 


47 


62 


109 


30 


53 


83 


Regression of 


Misses 


11 


31 


42 


34 


28 


62 


Raw Scores 


Total 


58 


93 


151 


64 


81 


145 


/ 

Multiple Linear 


Hits 


45 


56 


101 


38 


52 


90 


Regression of 


Misses 


13 


37 


50 


26 


29 


55 


Factor Scores 


Total 


58 


93 


151 


64 


81 


145 


Single 


Hits 


45 


50 


95 


44 


47 


91 


Best 


Misses 


13 


43 


56 


20 


34 


54 


Predictor 


Total 


58 


93 


151 


64 


81 


145 



*Success clients were those who were employed competitively or judges employable 
after evaluation. Unsuccessful clients (fail) were those judged as unemployable 
or as needing further rehabilitation services before they could be considered 
employable. 



3 

ERIC 

imiflaffHEaoaa 









- 5 - 



These points suggest that an investigator should consider several models 
of different levels of complexity when attempting to formulate a predic- 
tion scheme. In addition, crossvalidation of the prediction equations 
is necessary to assess their "true" predictive power. ' 

Even though these suggestions may be followed and the "best" of 
several models identified, the "best" model may still be inadequate 
for some purposes. For instance, the most adequate predictive model 

t 

in this study had a validity rate of 62.8%. Using this model one might 
expect to correctly predict about 63% of the client outcomes. Predic- 
tions for about 37% of the clients would be in error. This level of 

IT 

error would seem to be too great for most selection or placement deci- 
sions within a rehabilitation program. However, such a model might be 
used to identify clients who are predictably unsuccessful in a parti- 
cular rehabilitation program. Perhaps by intensive study of these 
client's characteristics one might identify new rehabilitation techniques 
to meet the special needs of this group. In addition, by studying the 
characteristics of those clients for whom predictions are in error, one 
might be able to develop new measurement instruments which would add to 
the accuracy of prediction models. 




m**M*\\m 



REFERENCES 



Ayer, M. J», Thoreson, R. , and Butler, A. Predicting rehabilitation 

success with the MMPI and demographic data. Personnel and Guidance 
Journal , 1966, 4-4, 631-637. 

BeMann, M. A predictive study of rehabilitation counseling outcomes. 
Journal of Counseling Psychology , 1963, ,10, 340-343. 

Drasgow, J., and Dreher, R. Predicting client readiness for training 
and placement in vocational rehabilitation. Rehabilitation Coun- 
seling Bulletin . 1964, 94-98. 

Helms tadter, G. Principles of psychological measurement . New York: 
Appleton-Century-Crofts , 1964. 

Novis, F., Marra, J., and Zadrozny, L. Quantitative measurement in the 
initial screening of rehabilitation potential. Personnel and 
Guidance Journal . 1960, 39 , 262-269. 

Perlman, L., and Hylbert, K. Identifying potential dropouts at a 
rehabilitation center. Rehabilitation Counseling Bulletin , 

1969, 13, 217-225. 

Veldman, D. Fortran programming for the behavioral sciences . New 
York: Holt, Rinehart, and Winston, 1967. 



